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ABSTRACT 



This paper discusses performance measurement in university 
libraries, based on examples from the University of Wollongong (UoW) in 
Australia. The introduction highlights the integration of information 
literacy into the curriculum and the outcomes of a 1998 UoW student 
satisfaction survey. The first section considers performance indicators in 
context, including the UoW's use of the Australian Quality Council’s 
"Australian Business Excellence Framework." The second section describes the 
development of performance indicators, including a framework that provides 
expectations/critical success factors and performance indicators for several 
stakeholders (i.e., university executive, clients, staff, suppliers, and 
community) . The third section addresses benchmarking, stressing the 
importance of benchmarking as a means of identifying best practice and of 
continuously improving all aspects of performance. The following uses of 
performance measurement are identified: to indicate the library or 
information services’ alignment with broader organizational goals; to 
demonstrate the integration of information services with the key activities 
of the organization or community; and to support the library’s position as 
the organization's primary information manager and service provider. (MES) 
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Introduction 



The announcement of the establishment of a Quality Audit Agency to evaluate the performance of 
universities, signalled an inevitable expansion of the incipient culture of measurement and 
evaluation in Australian universities. Those who consider that quality, and its associated tenets of 
measurement and evaluation are of dubious value, will be constrained to demonstrate goal 
achievement through whichever mechanisms are deemed appropriate by the Agency. Otherwise, 
they may face an uncertain funding future. Although the details of the audit process are not yet 
clear, there will be a single national body which will be responsible for overseeing independent 
external quality audits of universities’ internal systems. 

Universities will be asked to establish their own goals and conduct a self-assessment across the full 
range of their activities. The actual details of the audit process are still being negotiated. It is 
intended that all universities will be assessed every five years. Eventually, a university which fails 
to respond adequately to criticisms within the Agency reports could have its funding withdrawn. 

The full range of activities will assuredly include libraries, which are a significant investment for 
universities or, depending on your perspective, a huge drain on resources. Although self-evident to 
many, it has always been problematic to demonstrate the value of libraries. One of the main 
reasons is that it is difficult, if not impossible, to place a value on information - our main product. 

Libraries have been conscientious in producing statistical measures of collection size and the 
outputs of various internal processes. These have been useful, particularly when value was equated 
with collection size. While acknowledging that, in many instances, there is no substitute for 
significant, comprehensive on-site collections, technologically driven improvements in the 
distribution of, and access to, resources has seriously undermined the bigger is betters alue 
proposition. The perception that wanted information is ubiquitously and freely available and that 
libraries no longer have a vital role in universities has provided further impetus, in terms of future 
viability, for libraries to demonstrate that they are not only essential to the success of the ( 
university's researchers and students, but are of strategic importance in achieving the university's 
mission and goals. 

Proving that libraries are of strategic importance will not be simple, no matter how obvious it may 
appear to those in the profession and to many of our clients. Some possible approaches, based on 
examples from the University of Wollongong are given below. 

Closer involvement in University-level planning processes and in the development of courses in 
conjunction with faculties and media units, are strategies worth pursuing. Inclusion of 
library-related goals in statements of Graduate Attributes and University Strategic Plans are a first 
step. The Attributes of a Wollongong Graduate include: 

A basic understanding of information literacy and specific skills in acquiring, organising 
and presenting information, particularly through computer-based activity. 

The Library took a leading role in integrating information literacy into the curriculum and in 
developing and managing the compulsory program which is a zero credit point subject -ILIP100 - 
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for newly enrolling undergraduate students. One simple measure is successful completion rates for 
this subject, the assessment, however, of competency in the skills is problematic. Work conducted 
at Wollongong through its Tertiary Literacies Working Party, chaired by Lynne Wright, the Library's 
Client Services Manager, has concluded that there is . . .dear evidence to program participants of 
the extent to which particular competencies, skills and attributes are present (Temmerman & 
Wright 2000 p 7). 

Another graduate attribute acknowledges the need to inculcate skills for lifelong learning: 

A commitment to continued and independent learning, intellectual development, critical 
analysis and creativity. 

Chris Brewer, Health and Behavioural Sciences Faculty Librarian, has completed a study with our 
School of Nursing which aimed to evaluate the contribution of an integrated information literacy 
program to the inculcation of skills and student confidence in locating and using information 
(Brewer 1999). Although the statistical measures were inconclusive, qualitative data gained from 
focus groups and surveys indicated an appreciation of the value of information literacy programs to 
the development of study and research skills. 

Other sources of measurement of strategic importance include surveys exploring value or bivariate 
surveys which explore importance as well as performance. The following outcomes of the 1998 
UoW Student Satisfaction Survey indicate that the availability of library resources are of 
considerably greater importance than some other university services. 



Importance of Facilities 


VS% 


S% 


M% 


G% 


VG% 


Library - range of appropriate texts 


1 


9 


6.7 


29.2 


63.1 


Library - availability of appropriate texts 


2 


8 


6.8 


28.4 


63.8 


PG study room or office space 


2.1 


4.3 


23.9 


34.5 


35.1 


Laboratories 


6 


1.1 


14 


41 


43.2 


Lecture/tutorial facilities overall 


3 


5 


12.2 


43.1 


43.9 


Computer facilities 


5 


1.8 


11.7 


35.1 


50.9 


Importance: VS=very slight, S=slight, 


M=moderate, G= 


great, 


VG=very great 



The above examples are random and inconclusive and, alone, say little about organisational 
performance. 

Performance indicators in context 

Measurement and evaluation of library performance has been widely canvassed in the literature. 

For those seeking the most up-to-date assessment of performance measurement in Australian 
university libraries, the single most useful tool is the report of a DETYA EIP Project (Wilson 1999) 
and the Best Practice Handbook (Wilson & Pitman 1999), developed as an outcome of this project. 
(Note: the Council of Australian University Librarians intends to keep this groundbreaking work up 
to date through the development of a performance indicator website which will enabie practitioners 
to share data and methodologies). As demonstrated by the EIP Project Team's research, 
performance indicators and the application of benchmarking methodologies are the most commonly 
used tools for measuring performance. The most important consideration in selecting performance 
indicators is that they will enable you to measure what is critical to success in your environment. 

To quote from the Best Practice Handbook: 



1 p://www.alia.org.au/conferences/alla 2000 /proceedlngs/fellcity.mcgregor.html (2 of 7) [5/3/2001 9:08:05 AM] 



Felicity McGregor - Performance measures, benchmarking and value 

To be effective, performance indicators must be developed in context, not isolation. 

They must be firm/y rooted within a strategic management and planning framework ( 

Wilson 1999 p B3). 

The framework we have used since 1994 at Wollongong is the Australian Quality Council's 
Australian Business Excellence Framework. The introduction to the Framework states: This is 
Australia's Framework for innovation, improvement and long-term success, applicable to ait 
organisations, large and small, private and public, whatever their purpose (AQC 2000 pi). 

A large claim. However, in applying the principles underpinning the Framework over six years, I am 
able to affirm that the claims are not exaggerated and that the Framework has provided a solid 
foundation for the recognised success of the UoW Library. 

It has been statistically proven that the organisations that live by these principles demonstrate 
improved performance over the long term and provide examples of how creating best practices 
across the whole management system enhances positive outcomes for the whole organisation, all 
of its stakeholders and its society (AQC 2000 p5). 

Research underpinning this claim includes that conducted by Alexander Hausner, a postgraduate 
student of the University of Wollongong. Hausner's aim was to find whether quantitative evidence 
exists to link the Australian Business Excellence Framework (ABEF) with business outcomes. His 
research involved 22 manufacturing companies across a range of 13 different industry sectors, all 
of which had participated in the Australian Quality Awards for Business Excellence. Hausner's 
results included the following: 

The findings show a direct fink between performance in the Awards and annual 
improvement in bottom line results. (Hausner 1999, p I). 

Financial performance was not the only area of excellent performance: 

Management aspects such as senior executive leadership, analysis and use of data and 
information measures of success and planning processes were found to be of particular 
importance. 

It is concluded that striving for improvements against the ABEF is therefore in the 
interest of all stakeholders of an enterprise particularly the business owner and/or 
shareholder (Hausner 1999 p iv). 

Development of performance indicators 

If one accepts, from all of the above, that a measurement framework is desirable in improving and 
demonstrating organisational performance, then the choice of indicators and accompanying 
measures, which are valid, reliable and efficient to administer, remains a challenging process. 

- performance measurement is fundamentally multidimensional in nature. A library or information 
service that wishes to really understand how it is performing will examine both its environment and 
its constituencies - (Cullen 1999 p 26). 

At Wollongong, environmental analysis (through scenario development, SWOT analysis, 
professional reading and networks) has enabled us to develop Critical Success Factors - those key 
areas in which satisfactory results are essential to successful performance. 

We have also determined the expectations of library services of our various stakeholders (through 
surveys, focus groups, discussions). Our broad long term goals are derived from stakeholder 
expectations, from predictions about how expectations may change in future, analysis of 
opportunities in the professional and higher education environments, and from our own values and 
aspirations. 
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Together, the goals, critical success factors and stakeholder needs and expectations provide a basis 
for developing a system of measurement. These factors have been incorporated into a model 
known as a Performance Indicator Framework which links performance indicators and measures to 
each of the aforementioned elements. 



i 

Stakeholders j 


Expectations and Critical Success 
Factors 


Performance Indicators 


University Executive j 

\ 

1 

J 


leadership, cost efficiency; 
satisfaction of scholarly information 
needs, a model operation of quality 
processes. 


Leadership effectiveness, Effective 
budget utilisation, Client/stakeholder 
satisfaction, Bookvote use, 
benchmarking, external evaluation 


Clients j 

i 

i 

! 

i 

i i 


service excellence, flexible modes of 
access to resources and facilities to 
enable achievement of research, 
teaching and learning objectives. 


Collection relevance, Access to 
resources. Information Literacy level, 
Facilities use, Client satisfaction, 
Bookvote use 


| Staff | 

1 • • i 

. < 

i 

— _ __ <( j 


a safe and pleasant workplace, 
opportunity for career development, 
job security and empowerment in 
decision-making 


Leadership effectiveness. Staff 
development effectiveness, Staff 
satisfaction, Communication success, 
Skill levels, Workplace health and safety 


Suppliers 

i 

j 


mutual understanding of 
i requirements; the development of 
j innovative supply solutions, timely 
i supply of goods and services 


Supplier performance, Partnerships 


i Community 

! 

i 


j authoritative information resource, 
satisfaction of information needs, 
collaborative partnerships as well as 
the effective management of 
! resources 


Access to research resources, Community 
partnerships 



The framework incorporates both lead and /^indicators, /^indicators enable measurement of 
goal achievement and evaluation of success or assessment of benefits provided ovet a period of 
time. Lead indicators are used to measure process, operational performance and patterns of usage; 
to provide ongoing feedback and thus alert managers early to changes or unexpected variations. 

An example of a lag indicator is client satisfaction whereas a lead indicator may be client feedback 
incidents. 

All indicators at the UoW Library are designed to contribute to the evaluation of organisational 
performance through a single Key Performance Indicator: Client and Stakeholder Satisfaction. 

Performance indicators should be regularly reviewed, developed and refined as new sources of data 
emerge collection and analysis methodologies improve and the processes or performance to be 
evaluated change. Moreover, client and stakeholder expectations of library services ate constantly 
evolving in a dynamic environment and perceptions of value should be regularly surveyed to 
maintain alignment with internal operations and service priorities. 
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Value 



The question remains whether measures of organisational performance, strategic importance and 
stakeholder satisfaction can be aggregated to determine whether library and information services 
are of value to their clients. As mentioned in the introduction (above), it is difficult to measure the 
actual value of information with accuracy or objectivity. The question of value measurement is 
discussed extensively in a 1991 study by Broadbent and Lofgren who make the following points: 

Libraries support research and education which in themseives are difficult to assign 
precise economic value. (Broadbent 1991 p 96). 

In attempting objective measurements of the actual effect of information on performance, it is 
difficult to define and assess the impact of all the other variables which will affect the outcome. 

A library service may be a necessary prerequisite for the operation of a research department, but it 
is not likely to be possible to define the precise contribution of the library to research results ( ibid 
P 98). 

When the real impact of an information system cannot be measured, the perceived value may have 
to be accepted as a proxy. The perceived value approach is based on the subjective evaluation by 
users and presumes that users can recognise the benefits derived from an information service ... 
(ibid p 98). 

Acceptance of this position underpins our decision at Wollongong to adopt the one Key Performance 
Indicator of Client and Stakeholder Satisfaction. Measures, which include surveys, feedback 
incidents, timeliness, responsiveness and so on, are essentially measures of perception. Although 
research continues on identifying indicators which are objective and able to withstand academic 
scrutiny, there is a danger that the development, administration and interpretation of these 
indicators will be too time-consuming to be justified in an environment of resource constraints, and 
in which flexibility and change agility may be the key determinants of future sustainability. 

Agreement on the nature and purpose of performance indicators and the development of common 
instruments for measurement is also a prerequisite for the development of national standards or 
benchmarks. 

Benchmarking 



A system of regular internal measurement provides the information needed for benchmarking. 
Benchmarking has many definitions and many purposes. It is primarily a tool for learning and 
improvement. For organisations which aspire to excellence or best practice, then benchmarking is a 
means of testing achievements, services and processes against those of other organisations. 

Benchmarking thus needs not only to identify successes to date but also vita! signs of adaptation to 
the future. A university's dynamism is as important as its current achievements, indeed probably a 
better guide to its future performance McKinnon 2000 p 3). 

The Library profession is not lacking in data nor in performance indicators and measures. It does, 
however, lack clear benchmarks which can be used by- libraries to identify best practice, to 
constantly learn from each other and thus add value for their clients and to progressively improve 
the performance of all libraries and enhance their standing and recognition in the community. The 
recently published manual Benchmarking: a manual for Australian universities identifies 
benchmarks for all key areas of the University. The chapter on Library and Information Services is 
premised on the need to identify benchmarks which will assess Efficient use of resources and the 
quality of contributions to realisation of the university's objectives . . . (McKinnon 2000 p 115). 
Most of the benchmarks are criterion reference benchmarks, that is, they identify the attributes of 
good practice and can be used primarily for internal evaluation. To determine best practice, 
quantitative, comparable information would also be needed. 
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Benchmarking is also available to those libraries whose parent institution are members of the 
Commonwealth Higher Education Management Services (CHEMS) or Universitas 21, however, 
participation is limited to members and the benchmarks are not widely available. 

Those libraries participating in AQC Benchmarking networks are able to benchmark against network 
members - almost invariably non-library organisations - which in no way diminishes their 
usefulness but does not replace the need for specific library-related benchmarks. 

In conclusion, for reasons of both political necessity and future sustainability, as well as the need 
and desire of managers to know how their organisation is performing and whether library services 
are meeting the needs of clients and stakeholders in a timely, efficient and effective manner, 
performance measurement and benchmarking are essential tools. Benchmarking should not be 
construed as a competitive activity amongst libraries, rather as a means of identifying best practice 
and of continuously improving all aspects of performance. As a means of demonstrating 
competitiveness against non-library information providers, however, benchmarks could be 
extremely useful for libraries as they increasingly operate in an environment which is global in 
perspective and which is attracting potential competitors seeking to profit from the delivery of 
online education. 

Performance measurement is a highly political activity and must be seen as such, at the macro or 
micro level. We must look outwards to social and political expectations made of our institutions and 
ensure that they meet the needs and expectations of our significant client or stakeholder groups; 
we must use our planning and goal-setting activities in a meaningful way, incorporating 
appropriate measures, to demonstrate our response to this externa! environment, and our 
willingness to align our aspirations to broader corporate goals. But we must also look within and 
seek to promote an organisational culture which acknowledges the political nature of 
measurement. This means using performance measurement to: 

• Indicate the library or information service 's alignment with broader organisational goals; 

• Demonstrate the integration of information services with the key activities of the 
organisation, or of the community; 

• Support the library's position as the organisation's primary information manager and service 
provider (Cullen 1999 p 25). 
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